(1) Field of the Invention
The present invention relates to an image signal processing method for correcting distortions that occur in images shot at a set position and orientation of a camera, and modifying the distorted images into images which give more natural impressions to viewers, mainly in teleconference and videophone systems.
(2) Description of the Related Art
Since digital broadcasting and next-generation optical discs are now applicable to High-Definition Televisions (HDTV), large-sized screen display apparatuses such as HDTV-compatible Plasma Display Panels (PDP) displays and HDTV-compatible liquid crystal displays and the like have rapidly come into wide use. Further, high-speed network environment in which optical fiber is used has been popularized, enabling transmission and reception of data at a bit rate exceeding several Mbit/s even in common households. In the next few years, it is expected that transmission and reception of data at several dozen Mbit/s will be possible, and it is predicted that using image coding technology will accelerate the introduction of high-quality videophone and teleconference systems not only among companies with dedicated lines but also among general households.
When teleconferences are conducted and videophone calls are made, it is preferable to shoot front-view images of the speaker and send the same to the party at the other end. This is easily understood by analogy that in communication, talking with an eye contact gives a favorable impression on the other party and facilitates the communication. However, in real-life teleconferences and videophone calls, it is difficult to shoot front-view images of the speaker who often faces the display screen of the display apparatus.
FIG. 1 is a diagram showing exemplary positions of a display apparatus and a camera used in a teleconference system. A camera 101 is placed on a display apparatus 102. For shooting front-view images, it is preferable that the camera 101 is placed at the center of the display apparatus 102. However, the camera 101 placed at the center of the display apparatus 102 hinders a person from viewing images of the party at the other end displayed on the display apparatus 102. Consequently, it is common to place the camera 101 at the position shown in FIG. 1.
In recent years, the size of the display apparatus 102 has become larger, and large screen displays exceeding 40 inches in size are found even in households. The use of the display apparatus 102 with a large screen makes it easy to view the images of the party at the other end. However, on the other hand, the use of the display apparatus 102 with the large screen results in the position of the camera 101 placed on the display apparatus 102 to significantly shift from the front of the speaker.
FIGS. 2A, 2B, 2C, and 2D are diagrams showing an exemplary shot image in which a part that should be vertical is distorted when it is shot. The speaker stands upright in front of wall 104 which is perpendicular to the floor and poles 103 which support the wall 104. The camera 101 is placed on the display apparatus 102 and faces slightly downwards in order to shoot images of the speaker. FIG. 2A is a lateral view, taken from a lateral side of the speaker, showing how the camera 101 and the display apparatus 102 are placed. FIG. 2B is a front view taken from the front of the speaker. FIG. 2C is a top view taken from the top of the speaker.
Since the camera 101 is placed on the display apparatus 102, the distance between the camera 101 and the speaker, the distance between the camera 101 and the wall 104, and the distance between the camera 101 and the poles 103 significantly differ from each other depending on the position of the camera 101. In other words, the distance between the camera 101 and the bottom of the poles 103 (Lb) is noticeably longer than the distance between the camera 101 and the top of the poles 103 (Lt). It is well known that the images shot by cameras decreases in size with the distance from the subject. Thus, images shot by the camera 101 placed in the above described position are as shown in FIG. 2D. Comparison between FIG. 2B and FIG. 2D shows that FIG. 2D gives an unnatural impression on viewers because the bottom part (that is, the part which is more distant from the camera 101) of the image shown in FIG. 2D is displayed in smaller size and the poles 103 which should be originally vertical are shot at an angle.
The distortion attributable to the position of the camera 101 being not in front of the speaker can be corrected by calculation, given that the distances between the camera 101 and the subjects (that is, the speaker, the wall 104 and the poles 103 in this case) are provided. For example, given that the position and the orientation of the camera 101 are fixed, it is possible to calculate a method for correcting the images by measuring the distances in advance (See Japanese Patent No. 3109580). The camera disclosed in Japanese Patent No. 3109580 is a document camera, and it discloses measuring the distance between the camera 101 and the subject (the base where documents are placed) and the orientation of the camera 101, and correcting the shot image through image signal processing based on the measured distance and orientation.
However, with regard to teleconferences and videophones, there are cases where the position of the camera is shifted and the number of participants of the conference, the image shooting area and the like are frequently changed before the conference starts, which makes measurement of the distance between the camera 101 and the subject and measurement of the orientation of the camera 101 difficult. Consequently, there is a problem that image correction using the above described conventional technique cannot easily be implemented.
There is also a method considered for detecting image distortions by analyzing a shot image itself and correcting the detected distortions, instead of measuring a distance between the camera 101 and a subject and an orientation of the camera 101 and correcting images according to the measured distance and orientation. According to this method, there is an advantage that even when the position and the shooting angle and the like of the camera 101 are frequently changed, image corrections can be performed without being affected by such changes.
However, even with this method, a delay of at least one picture occurs in order to detect image distortions by analyzing the shot image. For this reason, there is a problem that a memory is necessary in order to hold image data of at least one picture.